Systems that make use of a mixed reality (MR) technique which naturally composites the real and virtual spaces have been extensively proposed. A typical MR system has the following arrangement. That is, an MR image is generated by aligning and compositing a virtual space image rendered by computer graphics (CG) to a real space image sensed by an image sensing device such as a camera or the like. The MR image is displayed on a display device such as a head-mounted display (HMD) that the user (observer/experiencer) wears, thus giving MR to the user.
In the field of virtual reality (VR), a technique that allows the user to manipulate (select, move, and so forth) a virtual object (an object on a virtual space) using a stylus (pen-like shaped pointing tool) as a real object has been prevalently known. The stylus as a real object has a sensor, which can measure the position and orientation of the stylus. Also, the stylus has a switch such as a button or the like. The user turns on/off the button to, e.g., select a virtual object pointed by the stylus.
In a normal MR system, upon generating an MR image, a virtual image that represents the virtual space is superimposed on a real image that represents the real space, and these images are composited. Hence, an image that always includes the virtual object in front of the real object is generated. When the user merely observes the MR image, no problem is posed. However, when the depth must be considered, e.g., when the user manipulates a virtual object by moving the stylus as a real object in his or her hand, a problem is posed.
That is, the stylus which is manually operated by the user often appears in the visual field of the image sensing device that senses the real space. Furthermore, the stylus is present at a depth position within about an accessible range of the user's hand. Therefore, when a virtual object is present outside the accessible range of the user's hand (farther than the stylus when viewed from the user), and the user makes some manipulation on that virtual object using the stylus, the stylus should be displayed in front of the virtual object, i.e., without being occluded by the virtual object. However, when a virtual image is merely composited on a real image, an MR image in which the stylus included in the real image is occluded by the virtual object is generated. The user who observes such MR image in which the depth ordering of the virtual object and real object (stylus) is reversed feels unnatural.
In order to solve such problem, a method of measuring the depth information of the real space including the stylus, and correctly expressing the depth ordering of the real and virtual spaces has been proposed. However, in order to precisely measure the depth information of real space in real time, an expensive apparatus such as a large-scale measuring device, multi-view stereo camera system, and the like must be used. For this reason, such method is often difficult to be adopted in terms of cost, thus calling improvements.
As a depth information acquisition method that can be implemented with relatively low cost, a method of obtaining depth information of the real space by applying an image process to real images sensed by a two-view stereo camera equipped on the HMD worn by the user is known. However, depth information obtained from two-view stereo camera images has low precision and resolution. Hence, such method is difficult to be adopted in an application that attaches importance on accurate expression of the depth ordering.
In order to solve these problems, the present applicant has proposed, in Japanese Patent Laid-Open No. 2003-296759 (US2003/0185461A1), a technique for correctly expressing the depth ordering by registering color information of a real object (object) to be displayed in front of a virtual image, extracting an object region from a real image using this color information, and inhibiting the virtual image from rendering on the object region. However, with this method, the color information required to detect the object region must be manually registered in the system. The manual registration process of the color information requires knowledge and skills, and is not a process that everyone can easily make.